Configuration

Your Beowulf cluster should have a NFS shared /home directory which is typically the way your cluster is configured it already configured. You will need at least 7 nodes to run the parallel version of the detection and tracking software called pmbarivision. There are also specific firewall and network settings required to run the pmbarivision through MBARI scripts noted in the instructions below.

Install Cluster Command Tool (optional)

If your cluster is configured with the Using the Open Source Cluster Application Resources (OSCAR), you already have this tool and you may skip this step.

As the root user, download and install Cluster Command Tool version 3.1 http://www.csm.ornl.gov/torc/C3/ (this also must be installed on all nodes in the cluster. see above site for more information). If nodes are removed or added to the cluster, you must update this list and restart the mpd service. This utility must be installed first to give you the ability to push files and execute commands easily on your worker nodes.

1. Download and install to /opt/c3-3.
2. Add a file /etc/c3.conf, e.g. for an 8-node cluster with master node named beowulffish
   /etc/c3.conf

   cluster oscar_cluster {
           beowulffish
           dead remove_line_for_0-indexing
           node1.private.net
           node2.private.net
           node3.private.net
           node4.private.net
           node5.private.net
           node6.private.net
           node7.private.net
           node8.private.net
   }

Install MPI libraries

The Message Passing MPI libraries are required to build pmbarivision. If you already have the MPI development libraries installed, then all that is required is to configure your mpd.hosts file (step 2).

1. As root user, download and install MPI libraries to /opt/mpich-2.1.1p1

   tar -zxvf mpich2-1.1.1p1.tar.gz
   cd mpich2-1.1.1p1
   ./configure --prefix=/opt/mpich-2.1.1p1
   make
   cd src/mpe2; make
   cd ..
   make install
   

2. Create and add the following to /opt/mpich2-1.1p1/etc/mpd.hosts

   touch /opt/mpich2-1.1p1/etc/mpd.hosts
   chmod a+r /opt/mpich2-1.1p1/etc/mpd.hosts
   

   /opt/mpich2-1.1p1/etc/mpd.hosts

   master.private.net
   node1.private.net
   node2.private.net
   node3.private.net
   node4.private.net
   node5.private.net
   node6.private.net
   node7.private.net
   node8.private.net

3. Push the libraries to the other nodes:

   cpush /opt/mpich-2.1.1p1
   

Add user aved

It is recommended to install and run the pmbarivision binary as a user aved that is visible across all nodes. To support this, create a user aved for installing and running pmbarivision. As the root user:

groupadd -g 300 aved
useradd -c "AVED" -g aved -u 300 aved

The aved user and group id must be synchronized across all nodes and machines in the Beowulf cluster. Once the aved user is created, syncing up the user is typically done automatically by the OSCAR Password Installer and User Management (OPIUM) software, but you can also do this manually with:

/opt/opium/bin/sync_users -f

Setup the user profile

Add to aved.sh/csh in the /etc/profile.d/ your MPDIR and SCRATCH_DIR settings. MPDIR points to the root of your mpich installation. SCRATCH_DIR is an NFS mounted directory that is shared between nodes where video will be processed. SCRATCH_DIR can be a separate disk, a separate partition, or a NFS shared directory - it just represents a tempporary "scratch" location that is shared across nodes where video will be temporarily placed for processing. SCRATCH_DIR must have read and write access for all users.

export MPDIR=/opt/mpich2-1.1.1p1
export PATH=$MPDIR/bin:$PATH
export SCRATCH_DIR=/mnt/scratch

set MPDIR = /opt/mpich2-1.1p1
set PATH = ($MPDIR/bin $PATH )
set SCRATCH_DIR = /mnt/scratch

When done push these to the other nodes:

cpush /etc/profile.d/aved.* /etc/profile.d/

Test password-less SSH across all nodes

Password-less ssh login is required to run the AVED MPI jobs. Your machine may already be configured for this. To test this, switch to the user aved and try ssh into the first node1. You should not be prompted for a password

root@beowulffish ~]# su aved
[aved@beowulffish ~]$ ssh n01
Last login: Thu Apr 15 10:54:27 2010 from master.private.net
[aved@node1 ~]$ 

If your machine is not configured for password-less login, there is a wealth of information on how to set this up on the web. However, this is typically configured by default in the OSCAR tool suite in /etc/profile.d/ssh-oscar.ssh/csh.

Copy ffmpeg library dependencies to all nodes

These dependencies are required in the pmbarivision code. This is the only dependency needed outside those installed in the /home/aved NFS shared directory. Push dependencies to the client nodes and refresh the dynamic linker:

cpush /usr/lib/libavcodec.so.51 
cpush /usr/lib/libavcodec.so.52 
cpush /usr/lib/libavcodec.so.51.40.4 
cpush /usr/lib/libavformat.so.51.12.1  
cpush /usr/lib/libavformat.so.51  
cpush /usr/lib/libavformat.so.51.12.1
cpush /usr/lib/libavutil.so
cpush /usr/lib/libavutil.so.49
cpush /usr/lib/libavutil.so.49.4.0 
cpush /usr/lib/libSDL-1.2.so.0
cpush /usr/lib/libSDL-1.2.so.0.7.0 
cexec 'ldconfig'